Software developers typically try to find and elimination sources of problems and failures of a software product during product testing that takes place as part of development of the software product. In some cases, testing of the software product may end there.
However, it is known to include error reporting services in computers to allow information on software problems to be collected. An example of such a service is the WINDOWS® Error Reporting Service provided by Microsoft Corporation of Redmond, Wash. When a failure, such as a crash or hang of an application occurs, the error reporting service may collect information about conditions on the computer leading up to the failure. This information, along with similar error reports from other computers executing the same application, may be sent to a central server, creating a database of failure that can be analyzed to identify software bugs that can be corrected.
Additionally, some software vendors may desire to refine and improve their software products continually, even if not in response to failures. Accordingly, it is also know to collect software quality metrics on the software product, while it is in use by customers. By collecting software quality metrics, it may be possible for software developers to identify ways a software product can be improved. For example, it may be advantageous to identify ways in which a software product is being used (e.g., frequency of use for specific features of the software product) or environments in which the software product is being used, and it may be advantageous to collect information on performance of the software product to optimize the software product for these uses or for execution in these environments.
Systems to collect software quality metrics are also known. As an example, the WINDOWS® operating system supports a software quality metric system that collects metrics relating to a software program from multiple different computers on which the software program executes. These metrics can relate to frequency and nature of use of software components, usability, reliability, and execution speeds of the software product. These metrics may be generated by the software program or may be calculated from raw instrumentation data output by a software product as it executes. Upon computation, the metrics may be transmitted to the central location for aggregation and analysis with metrics from other computing devices.
The metrics are collected and provided to the central location in accordance with instructions that are included in a released software product. For example, when a software product, or a new version of a software product, is released, the software product may include instrumentation metrics modules that are executed as a part of the software product. These metrics modules may include instructions, encoded as a part of the software product, that relate to collection of instrumentation data and computations on the instrumentation data to compute one or more metrics. When the software product is executed, the instructions of the metrics modules may be executed and may direct a computing device to collect specific instrumentation data, calculate instrumentation metrics based on that data, and provide the metrics to the central location.
As potential improvements are identified and made based on these metrics, and the software product is updated to a new version, the metrics modules and instructions may also be updated to direct the computing device to collect new and different instrumentation data and metrics to aid in identifying potential new sources of improvement.